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ABSTRACT 

The use of information and communication technologies (ICT) and therefore e-learning is becoming an ever more 
frequently used teaching and learning technique at all levels of education. In higher education, it completely or partially 
substitutes the classical teaching methods. It provides richer resources than the traditional classroom and overcomes its 
limitations (time and space). In some cases, it has proved to be very effective. Some authors, however, also point out that 
e-learning requires highly self-regulated and independent students or their e-learning performance may be low. The 
purpose of the paper is to analyse how introduction of the e-learning system Moodle as part of the teaching process is 
related with students’ performance, i.e. the average grade and the average number of admissions to the exams. We also 
examined the same relationship among different groups of students (based on selected individual socio-demographic 
factors). The study included a member of the University of Ljubljana (Faculty of Administration) with public 
administration programmes as our sample in the period from 2008 till 2014. The results of the analysis (using a t-test) 
show a significant improvement in performance (a higher average grade and lower average number of admissions) at the 
different (faculty-, student- and course-) levels after introducing the Moodle e-learning platform. The data show that the 
greatest improvement is seen among students with lower high school grades. The results can serve as important 
guidelines for university management when further investigating how to enhance students’ performance on different 
levels when employing modern ICT solutions in the teaching process. 
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1. INTRODUCTION 

In recent years, e-learning has been becoming increasingly interesting for society and educational institutions 
because it supports the concept of lifelong learning and since knowledge is becoming more and more 
important. This increases the demand for various educational forms and means. Different education 
programmes around the world cater to this increased demand and offer new forms of education that are 
frequently supported by information -communication technology (ICT) (Sulcic and Lesjak, 2009). Moreover, 
technological advances have revolutionized teaching and learning processes (Aristovnik, 2012). Fry (2001), 
for instance, notes that the emergence of new technologies, the rapid expiration of knowledge and training, 
the necessity of just-in-time information delivery, and the need for more cost-effective teaching methods 
have transformed teaching-learning practices. 

In comparison to traditional classroom instruction, the major advantages of e-learning are reducing 
geographical barriers as well as travel and programme overhead costs. The access becomes more flexible, 
from anywhere and usually at any time - essentially, it allows participants to collapse time and space (Cole, 
2000) - however, the learning materials must be designed properly to engage the learner and promote 
learning. On the other hand, there are also many problems that make e-learning ineffective. Many students 
drop out of online courses due to a lack of motivation, instructional design -related factors and learning style 
mismatch, time conflicts with other commitments, organizational support and follow-up on completion 
(Wang et ah, 2003). Other problems connected with e-learning are the absence of significant differences in 
acquired knowledge and the unsuccessfulness of e-leaming projects (Sulcic and Lesjak, 2009). 
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The effectiveness of e-learning systems has been an important subject of research in the last few decades. 
The opinions on the effectiveness of e-learning are many and very different. There are also many factors that 
influence the effectiveness of e-learning, with some being connected with technology/technics and others 
with people. Upadhyaya and Mallik (2013) claim that each issue of the effectiveness of e-learning cannot be 
treated as a standalone topic - either as a technical matter or as a people issue. E-learning involves interaction 
between people and processes, meaning that has to be treated as a socio-technical system rather than a social 
system considering only the people aspect (e.g. students, teachers and other stakeholders) or a technical 
system only considering the standards and processes aspect (e.g. course content, technology. Learning 
Management System (LMS), content management tools). E-learning is a complex process that depends not 
only on these aspects in isolation, but also the interaction among them. 

The purpose of the paper is to analyse how the introduction of the Moodle e-learning system as part of the 
teaching process is related to students’ performance in the public administration programmes, measured as 
the average grade and the average number of admissions to the exams for each course. We also examined the 
same relationship among different groups of students (based on gender, country region, high school grades) 
and courses (year of study, study programme, chair (faculty’s organizational unit) in which the course is 
conducted). The study included data for the period from 2008 till 2014. The paper explains how introduction 
of the Moodle e-learning system increased the students’ performance on the faculty level and discusses the 
variances among the different subgroups. 

The paper is structured as follows: first, a brief literature review about e-learning and its impact on 
students’ performance is presented. Second, we review the empirical methodology and describe the data. The 
third section outlines the empirical results and the discussion. The final section provides some concluding 
remarks, limitations and directions for future work. 


2. LITERATURE REVIEW 

Since e-learning has been an important and ever more frequently used teaching technique in the past decades, 
there are also many studies on its impact on students’ performance. Delivering instructions that can produce 
equal or even better outcomes than face-to-face learning systems is one of the main goals of introducing ICT 
into the study process (Saba, 2012). But besides the many advantages of this type of study, there are also 
many disadvantages which can reduce the positive impact of modem ICT tools on students’ performance. 
Moreover, in the e-learning process there are also many specific factors involved that are not directly 
connected with ICT but importantly influence the students’ performance, sometimes also in a negative way. 

Sulcic and Lesjak (2009) claim that, although ICT represents a fundamental material condition for 
e-Iearning, it does not have a statistically significant influence on the effectiveness of e-Iearning. Similarly, 
Russell (2001) discovered no statistically significant differences between classical and online learning. 
Different authors report varying attrition rates, e.g. from as high as 70%-80% (Dagger and Wade, 2004, 
Elood, 2002) to 20%-50% (Erankola, 2001; Diaz, 2002). These attrition rates can even be higher (between 
I0%-20%) than for traditional face-to-face education (Carr, 2000). When measuring students’ performance 
also with other measures, Novo-Corti, Varela-Candamio and Ramil-Diaz (2013) report an increase in the 
performance of students (grades and qualifications) when using the mixed technology of e-Iearning in 
Moodle and face-to-face lectures. 

According to Ally (2004), the teaching strategies as well as testing and assessment methods are much 
more important than the use of ICT in education. Students’ non-participation in the learning process plays an 
important role in high dropout rates in online learning (Dagger and Wade, 2004). It is therefore very 
important to determine which approaches might increase student engagement (Tyler-Smith, 2006). 

The review of the previous studies raises the dilemma of the validity of such comparisons since the 
statistics on retention and drop outs can be incomparable, unreliable and/or misleading (Hall, 2001; 
Haverilla, 2009; Saba, 2012; Wang, Eoucar-Szocki, Griffin, 2003). Researchers acknowledge that the reasons 
for attrition are many and complex and that there are no simple solutions (Berge and Huang, 2004). 

The same is true for students’ performance, measured as average grades and the average number of 
admissions to the exams. There are many factors that influence students’ performance, such as (Chien, 2012; 
Haverila and Barkhi, 2009; Kim and Kim, 2013; Saba, 2012; Upadhyaya and Mallik, 2013; Yukselturk and 
Burnt, 2007): learner characteristics (demographic factors, prior e-Iearning experience, ability and interest. 
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motivation, self-regulation), supporting systems (organizational aspects - e.g. structure and institutional 
arrangements; people aspects - e.g. motivation, training, other stakeholders) and e-learning system (quality 
of contents, tasks (instructor factors), technology - e.g. system reliability, user interface). 

It can be deduced from the reviewed literature that there is a huge research gap concerning the 
contributions of socio-demographic factors of students involved in an e-learning system to their academic 
performance. Nevertheless, a few researchers have focused on selected socio-demographic aspects of 
e-learning, such as Berge and Huang (2004), Yukselturk (2005) and Park and Choi (2009). In general, 
however, individual factors (i.e. socio-demographic, e.g. age, gender, education etc.) seem to have little 
influence on student performance. Indeed, Willging and Johnson (2004) claimed that external factors such as 
family issues, lack of organizational support, and workload are the crucial factors affecting students’ 
performance in an e-learning system (also see Park, 2007). Nevertheless, in our study we tried to fill above- 
mentioned research gap and highlight the importance of a selected socio-demographic background on 
(full-time) students’ performance in the Moodle environment in public administration programmes at the 
University of Ljubljana. 


3. MOODLE: E-LEARNING PLATEORM AT THE EACULTY OE 
ADMINISTRATION (UNIVERSITY OE LJUBLJANA) 

The open-source learning management system Moodle is already widespread in all segments of education, 
from primary schools to universities. It is free, flexible, customizable and basically contains many standard 
features which make it popular. It is available in more than 100 languages. Today, more than 54,000 active 
sites are registered from 225 countries, with millions of users (Moodle, 2015). Kareal and Klema (2006) 
compared particular features of some open-source e-learning systems and found that Moodle is one of the 
most adaptable, which is an essential part of effective education as they pointed out, and mots user-friendly 
learning systems among all the compared ones. Chen et al. (2011) studied the implementation of a Moodle 
course as an extension of a classical learning process at a university and found that for students “the Moodle 
e-learning platform is easy to use and provides a good communication tool, discussion area, group space, 
workspace, and makes learning more interesting”. 

The Faculty of Administration (FA) is part of the University of Ljubljana, Slovenia, that develops 
administrative science through research and education as well as integration into Slovenian and international 
practice. The FA offers two undergraduate study programmes - the University Study Programme in Public 
sector governance (US) and the Higher Education Ifrofessional Study Programme in Administration (PS) - 
and one joint undergraduate University Study Programme in Administrative Information Science (UI) 
together with the Faculty of Computer and Information Science. The undergraduate study programme (U' 
cycle) lasts 3 years (six semesters). Both undergraduate study programmes of the FA, the First Cycle 
Professional Study Programme in Administration and the First Cycle University Study Programme in Public 
Sector Governance, meet the high quality standards defined by the European Association for Public 
Administration Accreditation (EAPAA). The EA also offers the continuation of study in seven postgraduate 
programmes. The study programme is interdisciplinary, combining administrative, legal, economic as well as 
organizational and ICT courses. Consequently, the Eaculty is organized in three chairs: the Chair of 
Economics and Public Sector Management, the Chair of the Administrative-Legal Area and the Chair of 
Organization and Informatics. 

The EA began with blended learning in 2005 with eCampus, a payable platform for e-learning. By 
blended learning we mean the current use of the term, namely “combining Internet and digital media with 
established classroom forms that require the physical co-presence of teacher and students” (Eriesen, 2012). 
After three years of use, the learning platform was replaced with the open-source Moodle platform, mainly 
due to the user-friendly environment and cost benefits. 

At the beginning, the traditional teaching techniques were combined with an e-classroom on the basis of 
the voluntary decision of lecturers themselves. In the 2010/11 academic year, introduction of the Moodle 
e-learning system as part of the teaching process became mandatory for all courses in the first year of 
undergraduate study, namely 20 to 30 percent of the traditional face-to-face learning process was 
implemented in Moodle. In the next year, blended learning was implemented in the second year of study and 
in the 2012/13 academic year all undergraduate study courses had their own e-classroom in Moodle. 
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At the same time as implementing this combined method of learning, the rules (e-learning policies) on the 
quality of e-classrooms were determined. At the end of each semester, a consultant reviews the e-classrooms 
for e-studies and assesses them regarding their compliance with these rules. The quality assurance of the 
pedagogical work in e-classrooms in accordance with the defined rules is thus regularly monitored and 
necessary improvements are made. 


4. EMPIRICAL RESEARCH 
4.1 Methodology and Data 

In our study we analysed the performance of undergraduate students in two study programmes at the FA- the 
university (US) and the higher education professional study programme (PS), in the period between the 
2008/09 and 2013/14 academic years. We only included the compulsory courses for full-time students in our 
survey since the e-classroom for elective courses was completely mandatory. For each academic year, we 
surveyed all compulsory courses and checked what proportion of their realization (lectures, practical 
exercises) was held in Moodle. Using preliminary analyses, we defined that a course can be understood as 
(also) executed in Moodle if at least 1/5 of its realization is held in an e-classroom (i.e. a minimum 3 weeks 
in a semester of 15 weeks). 

The purpose of the research was to find out if the introduction of the blended learning with support of the 
Moodle platform had increased the students’ performance. We analysed and compared two indicators of 
students’ performance, namely students’ average grades and average required admissions to pass an exam in 
the years ‘with Moodle’ with the (previous) years without it. In the Slovenian higher education system, the 
grading scale ranges from 1 (minimum) to 10 (maximum) with 6 as a minimum passing grade. For our 
analysis, we used Student’s t-test for two independent samples. Table 1 shows the number of students 
enrolled in both programmes in each academic year between 2008/2009 and 2013/2014, where “Moodle - 
NO” means years with only face-to-face classrooms and “Moodle - YES” years with blended learning. 
Notice that in the last academic year all the courses are treated as courses with Moodle since it was 
mandatory for lecturers to have an e-classroom in Moodle. On the contrary, in the first two years (2008/09, 
2009/10) there are no courses treated as courses with Moodle. In the interim period, we have both types of 
courses: face-to-face and blended classroom. It is therefore possible that the same students were selected for 
both the “Moodle - NO” and “Moodle - YES” samples since they attended both types of courses. Notice that 
we only analysed obligatory courses so the students could not choose to be assigned to the “Moodle - NO” 
and “Moodle - YES” samples since they had to follow the lecturers’ decision. 

Table 1. Students enrolled in each academic year - in study years of both programmes (PS - Professional Study 
Programme and US - University Study Programme) 


Academic year 

fSt 

Moodle - NO 

2nd ^rd 

fSt 

Moodle -YES 

2nd ^rd 

2008/09 

PS 

213 

0 

0 

0 

0 

0 


US 

278 

0 

0 

0 

0 

0 

2009/10 

PS 

114 

0 

0 

0 

0 

0 


US 

208 

210 

0 

0 

0 

0 

2010/11 

PS 

157 

0 

96 

133 

0 

0 


US 

163 

191 

138 

148 

175 

0 

2011/12 

PS 

119 

0 

0 

156 

0 

89 


US 

108 

140 

147 

109 

0 

0 

2012/13 

PS 

0 

0 

0 

133 

0 

65 


US 

92 

103 

131 

0 

0 

0 

2013/14 

PS 

0 

0 

0 

no 

0 

67 


US 

0 

0 

0 

93 

80 

82 


Source: Survey, 2014 
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As explained in the above literature review section, students’ performance is influenced by many factors. 
We decided to eliminate the impact of (1) the fluctuation of lecturers and (2) different generations of 
students. Therefore, we only analysed courses where the same lecturer conducted a course without Moodle 
and later with it, both for at least two consequent years. This means we excluded all courses where the 
lecturer was changed. We also excluded courses where only one generation used Moodle or only one 
generation did not use it. Our original data table was therefore reduced to only 14 compulsory courses. 


4.2 Empirical Results 

On the faculty level, the introduction of Moodle is related to a significantly increased student performance. 
The average grade rose from 6.98 to 7.11 (Table 2), making the difference of 0.13 points highly significant 
(p-value: 2.7 lE-7). The decrease in the average number of admissions needed for the exams is even stronger 
(Table 3) - it went down from 1.73 admissions (Moodle - NO) to 1.30 (Moodle - YES). The difference of - 
0.43 is even more significant (p-value: 7.26E-86). 

Although the introduction of the Moodle platform at the faculty level showed a significant improvement, 
we assumed there might still be subgroups of students or courses where the improvement is not significant. 
Eor a more detailed analysis, we collected additional data on courses and students (factors). We added data 
about the study programme (US/PS), the year of study (E', 2"'*, 3'“^) and the chair (organizational unit at the 
Eaculty) to which a lecturer belongs. Erom the students’ enrolment form we extracted data on the students’ 
gender and the region of Slovenia where they were born. We only considered two ‘regions’ - “Central 
Slovenia” where the faculty is located - and all other regions in Slovenia as one region (“Other Slovenian 
regions”). In addition, we took the students’ high school final grade into account (four categories). 

Table 2. Average grades for years with Moodle (Moodle - YES) and years without it (Moodle - NO) among different 
factors (study programme, year of study, chair of lecturer; gender, region and high school final grade) with differences 

and corresponding p-values (Sig.) 



Moodle 
NO YES 

Difference 

Slg. 


Programme 

University Study Programme 

6.99 

7.12 

0.13 

3.18E-06 

*** 

Professional Study Programme 

6.93 

7.08 

0.15 

2.09E-03 


Year of study 

E‘ 

7.09 

7.20 

0.11 

6.80E-04 

* 

2nd 

6.82 

6.99 

0.17 

4.06E-05 

*** 

3rd 

6.89 

6.94 

0.05 

2.40E-01 


Chair 

Economics and Public Sector Management 

6.79 

7.06 

0.28 

4.88E-15 

*** 

Administrative-Legal Area 

7.24 

7.37 

0.13 

l.lOE-02 


Organization and Informatics 

6.91 

6.96 

0.05 

1.69E-01 


Gender 

Male 

6.86 

7.00 

0.14 

1.79E-03 


Female 

7.01 

7.15 

0.13 

6.21E-06 

*** 

Region 

Central Slovenia 

6.92 

7.10 

0.17 

8.35E-07 

*** 

Other Slovenian regions 

7.03 

7.13 

0.10 

3.17E-03 


High schooi final grade 

sufficient (2) 

6.89 

7.01 

0.12 

1.46E-05 

*** 

good (3) 

7.11 

7.17 

0.06 

1.61E-01 


very good (4) 

7.41 

7.45 

0.04 

3.43E-01 


excellent (5) 

7.61 

7.61 

0.00 

5.01E-01 


Together (at the faculty level) 

6.98 

7.11 

0.13 

2.71E-07 



Difference is significant at the following levels: 0.1 (*), 0.05 (**), 0.01 (***) 
Source: Survey, 2014 
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For each factor we calculated the average grade and the average number of admissions to the exams for 
all of their levels separately for the years with Moodle and the years without it. In Table 2 and Table 3 we 
report the averages and differences with corresponding p-values calculated with an independent samples 
t-test. We corrected the p-values with a Bonferroni correction and marked the significant ones with stars. The 
data in Table 2 indicate a highly significant increase in the average grade among the majority of categories. 

The biggest increase in the average grade belongs to the courses in the Chair of Economics and Public 
Sector Management. The average grade rose from 6.79 to 7.06, with the difference of 0.28 points being 
highly significant (p-value: 4.88E-15). We also discovered highly significant increases for students from 
Central Slovenia (increase from 6.92 to 7.10, p-value: 8.35E-7), students from the University programme 
(from 6.99 to 7.12, p-value: 3.18E-60), female students (from 7.01 to 7.15, p-value: 6.21E-6), students with 
high school grades 2 out of 5 (from 6.89 to 7.01, p-value: 1.46E-5) and students in the 2"‘* year of study in 
both study programmes (from 6.82 to 6.99, p-value: 4.06E-5). In the other subgroups, the increase in the 
average grade is not significant. On the other hand, it is interesting that the average grade went up in all 
subgroups with the only exception of students with an excellent high school background (grade 5 out of 5). 
We found at least one subgroup with a highly significant increase in the average grade for each factor 
(programme, year, chair, gender, region, high school grades) which helps us identity where the 
implementation of Moodle seems to play an important role in achieving high students’ average grades. 

Table 3 shows a highly significant decrease in the average number of admissions to the exams among all 
categories. A slight exception is the factor ‘high school final grade’. 

Table 3. Average number of admission for years with Moodle (Moodle - YES) and years without it (Moodle - NO) 
among different factors (study programme, year of study, chair of lecturer; gender, region and high school final grade) 

with differences and corresponding p-values (Sig.) 



Moodle 





NO 

YES 

Difference 

Sig. 


Programme 

University Study Programme 

1.76 

1.32 

-0.44 

3.40E-61 

*** 

Professional Study Programme 

1.58 

1.23 

-0.35 

9.16E-21 

*** 

Year of study 

U‘ 

1.72 

1.20 

-0.52 

1.05E-75 

*** 

2nd 

1.76 

1.43 

-0.33 

2.63E-15 

*** 

3rd 

1.66 

1.44 

-0.22 

2.37E-04 

*** 

Chair 

Economics and Public Sector Management 

1.76 

1.27 

-0.49 

8.16E-58 

*** 

Administrative-Legal Area 

1.43 

1.25 

-0.17 

7.17E-07 

*** 

Organization and Informatics 

2.02 

1.42 

-0.60 

4.44E-28 

*** 

Gender 

Male 

1.78 

1.32 

-0.45 

6.13E-23 

*** 

Female 

1.71 

1.29 

-0.43 

1.90E-65 

*** 

Region 

Central Slovenia 

1.75 

1.32 

-0.44 

2.81E-43 

*** 

Other Slovenian regions 

1.70 

1.27 

-0.43 

3.21E-41 

*** 

High schooi final grade 

sufficient (2) 

1.80 

1.32 

-0.47 

1.17E-63 

*** 

good (3) 

1.60 

1.28 

-0.32 

4.20E-13 

*** 

very good (4) 

1.40 

1.21 

-0.19 

5.89E-04 

* 

excellent (5) 

1.43 

1.13 

-0.30 

2.32E-03 


Together (at the faculty level) 

1.73 

1.30 

-0.43 

7.62E-86 



Difference is significant at the following levels: 0.1 (*), 0.05 (**), 0.01 (***) 
Source: Survey, 2014 
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Students with an excellent (grade 5 out of 5) and very good (grade 4 out of 5) high school background 
saw a decreased average number of admissions. The drops from 1.43 to 1.13 (excellent students) and from 
1.40 to 1.21 (very good students) are, however, too small to be significant. We can see that those students 
with a better high school background (grades) need fewer admissions than the students with lower high 
school grades. It is, however, encouraging that the Moodle environment helped students with a lower high 
school background reduce the average number of their admissions. Students with sufficiently high school 
grades (grade 2 out of 5) decreased the average number of admissions from 1.80 to 1.32. Although the 
decrease is significant for all factors, the data in Table 3 show a very significant decrease in the average 
number of admissions for the T'’* year of study (from 1.72 to 1.20 with p-value; 1.05E-75) and female 
students (from 1.71 to 1.29 with p-value: 1.90E-75). We can also find highly significant differences in the 
University Programme and courses from the Chair of Economics and Public Sector Management. 


5. CONCLUSION 

The results of our study indicate that the implementation of an e-learning system (Moodle) at the Eaculty of 
Administration, University of Ljubljana is related to a statistically significant increase in students’ 
performance, measured as the average grade and the average number of admissions to the exams. We 
demonstrated that the courses from the Chair of Economics and Public Sector Management, female students 
and students with a lower high school background benefitted more from the work in the Moodle environment 
than the other groups of studied entities. Based on our empirical results, we can also conclude that almost all 
subgroups of courses and students reduced the average number of admissions to the exams and increased the 
average grades after the Moodle platform was introduced. Interestingly, the only subgroup of students which 
did not benefit much is the group of students with the highest grades from high school. 

The main limitation of the research is the limited data set (i.e. the reduction of the number of courses 
analysed due to the lecturers’ fluctuation and different generations of students). Besides that, we did not take 
into account other individual factors (e.g. motivation) and external factors (e.g. contents quality, previous 
trainings in Moodle, instructor factors, technological characteristics of the Moodle platform). These factors 
should be the subject of our further research in the near future. Moreover, future research could also broaden 
the scope of the current study, which was restricted to the public administration programme. Einally, 
additional studies should seek to analyse the data from different study programmes in order to increase the 
validity of results for the entire University of Ljubljana. To conclude, the study results can still serve as 
important background material when deciding on the future development of e-learning at the Eaculty of 
Administration as well as on the introduction of e-learning platforms at other faculties within the University 
of Ljubljana. The empirical results pointed out the main challenge: how to use the Moodle platform to 
increase the grades of students with the best high school backgrounds and therefore to increase the graduates’ 
capabilities to solve challenges in public administration. 
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